Safe Screening with Variational Inequalities and Its Application to Lasso

We begin with three technical lemmas. Lemma 2 Let y ̸ = 0 and 0 < λ 1 ≤ ∥X T y∥ ∞. We have ⟨ y λ 1 − θ * 1 , θ * 1 ⟩ ≥ 0. Proof Since the Euclidean projection of y λ1 onto {θ : ∥X T θ∥ ∞ ≤ 1} is θ * 1 , it follows from Lemma 1 that ⟨θ * 1 − y λ 1 , θ − θ * 1 ⟩ ≥ 0, ∀θ : ∥X T θ∥ ∞ ≤ 1. (46) As 0 ∈ {θ : ∥X T θ∥ ∞ ≤ 1}, we have Eq. (45). Lemma 3 Let y ̸ = 0 and 0 < λ 1 ≤ ∥X T y∥ ∞. If θ * 1 parallels to y in that it can be written as θ * 1 = γy for some γ, then γ = 1 ∥X T y∥∞. Proof Since y ∥X T y∥∞ satisfies the condition in Eq. (11), we have ⟨γy − y λ 1 , y ∥X T y∥ ∞ − γy⟩ = (γ − 1 λ 1)(1 ∥X T y∥ ∞ − γ)∥y∥ 2 2 ≥ 0 (47) which leads to γ ∈ [ 1 ∥X T y∥∞ , 1 λ1 ]. In addition, since ∥X T θ * 1 ∥ ∞ ≤ 1, we have γ = 1 ∥X T y∥∞. This completes the proof. Lemma 4 Let y ̸ = 0. If 0 < λ 1 ≤ ∥X T y∥ ∞ , we have ⟨ y λ 1 − θ * 1 , y⟩ ≥ 0, (48) where the equality holds if and only if λ 1 = ∥X T y∥ ∞. where the equality holds if and only if y λ1 = θ * 1. Incorporating Eq. (45) in Lemma 2 and Eq. (49), we have Eq. (48). The equality in Eq. (49) holds if and only if y λ1 = θ * 1. According to Lemma 3, if θ * 1 = y λ1 , then θ * 1 = y ∥X T y∥∞ , which leads to λ 1 = ∥X T y∥ ∞. This ends the proof. Now, we are ready to prove Theorem 1. If follows from Eq. (17) and Eq. (48) ⟨b, a⟩ = (1 λ 2 − 1 λ 1)⟨ y λ 1 − θ * 1 , y⟩ + ∥ …